• PROJECT OBJECTIVE: To understand K-means Clustering by applying on the Car Dataset to segment the cars into various categories.¶
In [146]:
import numpy as np
import pandas as pd
import json
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.model_selection import train_test_split
from sklearn.cluster import KMeans
1. Data Understanding & Exploration:¶
1.A. Read ‘Car name.csv’ as a DataFrame and assign it to a variable
In [147]:
d1= pd.read_csv("C:\\Users\\HARITHA\\Car name.csv")
In [148]:
d1.head()
Out[148]:
1.B. Read ‘Car-Attributes.json as a DataFrame and assign it to a variable. [
In [149]:
d2 = pd.read_json("Car-Attributes.json")
In [150]:
d2.head()
Out[150]:
1.C. Merge both the DataFrames together to form a single DataFrame
In [151]:
df=d1.join(d2)
In [152]:
df.head()
Out[152]:
In [153]:
df.shape
Out[153]:
1.D. Print 5 point summary of the numerical features and share insights.
In [154]:
df.describe().T
Out[154]:
- insights:
In [155]:
sns.boxplot(df,orient='h');
2. Data Preparation & Analysis:¶
2.A. Check and print feature-wise percentage of missing values present in the data and impute with the best suitable approach.
In [156]:
percent_missing = df.isnull().sum() * 100 / len(df)
In [157]:
percent_missing
Out[157]:
In [158]:
df.isnull().sum()
Out[158]:
2.B. Check for duplicate values in the data and impute with the best suitable approach
In [159]:
df.duplicated().sum()
Out[159]:
2.C. Plot a pairplot for all features.
In [160]:
sns.pairplot(df,diag_kind='kde');
2.D. Visualize a scatterplot for ‘wt’ and ‘disp’. Datapoints should be distinguishable by ‘cyl’.
In [161]:
sns.scatterplot(data=df,x='wt',y='disp',hue='cyl',palette='dark');
2.E. Share insights for Q2.d
- if the wt increases the disp increases.
2.F. Visualize a scatterplot for ‘wt’ and ’mpg’. Datapoints should be distinguishable by ‘cyl’
In [162]:
sns.scatterplot(data=df,x='wt',y='mpg',hue='cyl',palette='bright');
2.G. Share insights for Q2.f
- when the wt increases mpg gets decreased, if the wt is low the mpg will be high
2.H. Check for unexpected values in all the features and datapoints with such values.
In [163]:
df['hp'].sample(25)
Out[163]:
In [164]:
df[df['hp']=="?"]
Out[164]:
In [165]:
df['hp'].replace("?",np.nan, inplace=True)
In [166]:
df['hp'].iloc[336]
Out[166]:
In [167]:
df['hp'].dropna(inplace=True)
In [168]:
df['hp'].iloc[336]
Out[168]:
In [169]:
df.shape
Out[169]:
In [170]:
df.info()
In [171]:
df['hp'].fillna((df['hp'].median()), inplace=True)
df['hp'] = df['hp'].astype('float')
In [172]:
df.dtypes
Out[172]:
In [173]:
# importing the StandardScaler Module
from sklearn.preprocessing import StandardScaler
In [174]:
# Creating an object for the StandardScaler function
X = StandardScaler()
In [175]:
scaled_df = X.fit_transform(df.iloc[:,1:9])
In [176]:
scaled_df
Out[176]:
3. Clustering:¶
3.A. Apply K-Means clustering for 2 to 10 clusters.
In [177]:
df.isnull().sum()
Out[177]:
In [178]:
df.dropna(inplace=True)
In [179]:
df.isnull().sum()
Out[179]:
In [180]:
df.groupby(df['cyl']).mean()
Out[180]:
In [181]:
# Calculate age of vehicle
df['age'] = 83-df['yr']
df.head()
Out[181]:
In [182]:
#Convert origing into dummy variables (This again is subjected to business knowledge. We might drop this variable as well
# Inclusion is more to demonstrate on how to use categorical data)
one_hot = pd.get_dummies(df['origin'])
one_hot = one_hot.add_prefix('origin_')
# merge in main data frame
df = df.join(one_hot)
df.head()
Out[182]:
In [183]:
df_new = df.drop(['yr','origin','car_name'], axis =1)
df_new.head()
Out[183]:
In [184]:
sns.boxplot(data=df_new);
In [185]:
# We could see some outliers for mpg,hp and acc
sns.boxplot(y=df_new['mpg']);
In [186]:
sns.boxplot(y=df_new['hp']);
In [187]:
df_new['hp'] = np.log(df_new['hp'])
df_new['acc'] = np.log(df_new['acc'])
df_new['mpg'] = np.log(df_new['mpg'])
df_new.head()
Out[187]:
In [188]:
sns.boxplot(data=df_new);
In [189]:
from scipy.stats import zscore
df_new.dtypes
numeric_cols = df_new.select_dtypes(include=[np.int64, np.float64]).columns
numeric_cols
df_new[numeric_cols] =df_new[numeric_cols].apply(zscore)
In [190]:
df_new.head()
Out[190]:
In [191]:
cluster_range = range(2,11)
cluster_errors = []
for num_clusters in cluster_range:
clusters = KMeans(num_clusters, n_init = 5)
clusters.fit(df_new)
labels = clusters.labels_
centroids = clusters.cluster_centers_
cluster_errors.append(clusters.inertia_)
clusters_df = pd.DataFrame({"num_clusters": cluster_range, "cluster_errors": cluster_errors})
clusters_df
Out[191]:
3.B. Plot a visual and find elbow point
In [192]:
from matplotlib import cm
plt.figure(figsize=(12,6))
plt.plot( clusters_df.num_clusters, clusters_df.cluster_errors, marker = "o" )
Out[192]:
3.C. On the above visual, highlight which are the possible Elbow points
- Based on the above visual the possible elbow points would be 3,4,5
3.D. Train a K-means clustering model once again on the optimal number of clusters
In [193]:
#Set the value of k=4
kmeans = KMeans(n_clusters=4, n_init = 15, random_state=2345)
In [194]:
kmeans.fit(df_new)
Out[194]:
In [195]:
centroids = kmeans.cluster_centers_
In [196]:
centroids
Out[196]:
In [197]:
#Calculate the centroids for the columns to profile
centroid_df = pd.DataFrame(centroids, columns = list(df_new) )
In [198]:
print(centroid_df)
In [199]:
sns.scatterplot(centroid_df,legend='full')
Out[199]:
3.E. Add a new feature in the DataFrame which will have labels based upon cluster value.
In [200]:
## creating a new dataframe only for labels and converting it into categorical variable
df_labels = pd.DataFrame(kmeans.labels_ , columns = list(['labels']))
df_labels['labels'] = df_labels['labels'].astype('category')
In [201]:
# Joining the label dataframe with the data frame.
df_labeled = df.join(df_labels)
In [202]:
df_analysis = (df_labeled.groupby(['labels'] , axis=0)).head(4177) # the groupby creates a groupeddataframe that needs
# to be converted back to dataframe.
df_analysis
Out[202]:
In [203]:
df_labeled['labels'].value_counts()
Out[203]:
3.F. Plot a visual and color the datapoints based upon clusters
In [204]:
from mpl_toolkits.mplot3d import Axes3D
In [205]:
## 3D plots of clusters
fig = plt.figure(figsize=(8, 6))
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=20, azim=60)
kmeans.fit(df_new)
labels = kmeans.labels_
ax.scatter(df_new.iloc[:, 0], df_new.iloc[:, 1],df_new.iloc[:,9],c=labels.astype(np.float), edgecolor='k')
ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])
ax.set_xlabel('Length')
ax.set_ylabel('Height')
ax.set_zlabel('Weight')
ax.set_title('3D plot of KMeans Clustering')
Out[205]:
3.G.Pass a new DataPoint and predict which cluster it belongs to
In [206]:
df_row={'car_name': 'ford torino',
'mpg': '17.0',
'cyl': '80',
'disp': '303.0',
'hp': '145.0',
'wt': '3431',
'acc': '11.5',
'yr': '80',
'origin':'1',
'age': '13',
'origin_1': '1',
'origin_2': '0',
'origin_3':'0'}
In [207]:
df= df.append(df_row,ignore_index=True )
In [208]:
df.shape
Out[208]:
In [297]:
sns.scatterplot(data=df_new,legend='auto');
PART B :¶
1. Data Understanding & Cleaning:¶
1.A. Read ‘vehicle.csv’ and save as DataFrame.
In [210]:
#For numerical libraries
import numpy as np
#To handle data in the form of rows and columns
import pandas as pd
#importing seaborn for statistical plots
import seaborn as sns
#importing ploting libraries
import matplotlib.pyplot as plt
#styling figures
plt.rc('font',size=14)
sns.set(style='white')
sns.set(style='whitegrid',color_codes=True)
#To enable plotting graphs in Jupyter notebook
%matplotlib inline
#importing the Encoding library
from sklearn.preprocessing import LabelEncoder
#Build the model with the best hyper parameters
from sklearn.model_selection import cross_val_score
#importing the zscore for scaling
from scipy.stats import zscore
#Importing PCA for dimensionality reduction and visualization
from sklearn.decomposition import PCA
# Import Logistic Regression machine learning library
from sklearn.linear_model import LogisticRegression
# Import Support Vector Classifier machine learning library
from sklearn.svm import SVC
#Import Naive Bayes' machine learning Library
from sklearn.naive_bayes import GaussianNB
#Import Sklearn package's data splitting function which is based on random function
from sklearn.model_selection import train_test_split
#Grid search to tune model parameters for SVC
from sklearn.model_selection import GridSearchCV
# Import the metrics
from sklearn import metrics
In [211]:
vehicle_df=pd.read_csv('C:\\Users\\HARITHA\\vehicle.csv')
In [212]:
vehicle_df.head(10)
Out[212]:
In [213]:
vehicle_df.info()
1.B. Check percentage of missing values and impute with correct approach
In [214]:
vehicle_df.isnull().sum()
Out[214]:
In [215]:
percent_missing = vehicle_df.isnull().sum() * 100 / len(df)
In [216]:
percent_missing
Out[216]:
In [217]:
#class attribute is not an object it is a category
vehicle_df['class']=vehicle_df['class'].astype('category')
In [218]:
#To get the shape
vehicle_df.shape
Out[218]:
In [219]:
#To get the number of columns
vehicle_df.columns
Out[219]:
In [220]:
#Checking for missing values in the dataset
vehicle_df.isnull().sum()
Out[220]:
In [221]:
#replace missing variable('?') into null variable using numpy
vehicle_df = vehicle_df.replace(' ', np.nan)
In [222]:
#Replacing the missing values by median
for i in vehicle_df.columns[:17]:
median_value = vehicle_df[i].median()
vehicle_df[i] = vehicle_df[i].fillna(median_value)
In [223]:
# again check for missing values
vehicle_df.isnull().sum()
Out[223]:
In [224]:
# Again check data information
vehicle_df.info()
In [225]:
# Understand the spread and outliers in dataset using boxplot
vehicle_df.boxplot(figsize=(35,15));
In [226]:
# Histogram
vehicle_df.hist(figsize=(15,15));
In [227]:
#find the outliers and replace them by median
for col_name in vehicle_df.columns[:-1]:
q1 = vehicle_df[col_name].quantile(0.25)
q3 = vehicle_df[col_name].quantile(0.75)
iqr = q3 - q1
low = q1-1.5*iqr
high = q3+1.5*iqr
vehicle_df.loc[(vehicle_df[col_name] < low) | (vehicle_df[col_name] > high), col_name] = vehicle_df[col_name].median()
In [228]:
# again check for outliers in dataset using boxplot
vehicle_df.boxplot(figsize=(35,15));
In [229]:
print('Class: \n', vehicle_df['class'].unique())
In [230]:
vehicle_df['class'].value_counts()
Out[230]:
In [231]:
#Encoding of categorical variables
labelencoder_X=LabelEncoder()
vehicle_df['class']=labelencoder_X.fit_transform(vehicle_df['class'])
In [232]:
#correlation matrix
cor=vehicle_df.corr()
cor
Out[232]:
In [233]:
# correlation plot---heatmap
sns.set(font_scale=1.15)
fig,ax=plt.subplots(figsize=(18,15))
sns.heatmap(cor,vmin=0.8, annot=True,linewidths=0.01,center=0,linecolor="white",cbar=False,square=True)
plt.title('Correlation between attributes',fontsize=18)
ax.tick_params(labelsize=18)
In [234]:
#pair panel
sns.pairplot(vehicle_df,hue='class');
1.C. Visualize a Pie-chart and print percentage of values for variable ‘class’
In [235]:
vehicle_df['class'].value_counts()
Out[235]:
In [236]:
per_class=vehicle_df['class'].value_counts() * 100 / len(vehicle_df['class'])
In [237]:
per_class
Out[237]:
In [238]:
plt.pie(per_class, autopct='%1.1f%%');
1.D. Check for duplicate rows in the data and impute with correct approach.
In [239]:
vehicle_df.duplicated().sum()
Out[239]:
2. Data Preparation:¶
2.A. Split data into X and Y. [Train and Test optional]
In [240]:
#independent and dependent variables
X=vehicle_df.iloc[:,0:18]
y = vehicle_df.iloc[:,18]
In [241]:
# Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 10)
In [242]:
model = LogisticRegression()
model.fit(X_train, y_train)
prediction = model.predict(X_test)
In [243]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ',model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
- The accuracy, recall, and precision values using Logistic Regression model is high without applying dimentionality reduction (PCA).
In [244]:
resultsDf=pd.DataFrame({'Model':['Logistic'],'Accuracy': model.score(X_test , y_test)},index={'1'})
resultsDf=resultsDf[['Model','Accuracy']]
resultsDf
Out[244]:
In [245]:
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X_train, y_train)
prediction = model.predict(X_test)
In [246]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
In [247]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['Naive Bayes'], 'Accuracy': model.score(X_test, y_test)},index={'2'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[247]:
In [248]:
clf = SVC()
clf.fit(X_train, y_train)
prediction = model.predict(X_test)
3.B. Print Classification metrics for train data.
In [249]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
In [250]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['SVM'], 'Accuracy': model.score(X_test, y_test)},index={'3'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[250]:
3.C. Apply PCA on the data with 10 components.
In [251]:
# Scaling the independent attributes using zscore
X_z=X.apply(zscore)
In [252]:
# prior to scaling
plt.rcParams['figure.figsize']=(10,6)
plt.plot(vehicle_df)
plt.show()
In [253]:
#plt.plot(X_z,figsize=(20,10))
plt.rcParams['figure.figsize']=(10,6)
plt.plot(X_z)
plt.show()
In [254]:
# Calculating the covariance between attributes after scaling
cov_matrix = np.cov(X_z.T)
print('Covariance Matrix \n%s', cov_matrix)
In [255]:
#Finding eigenvalues amd eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s', eigenvectors)
print('\n Eigen Values \n%s', eigenvalues)
In [256]:
# Make a set of (eigenvalue, eigenvector) pairs
eigen_pairs = [(np.abs(eigenvalues[i]), eigenvectors[:,i]) for i in range(len(eigenvalues))]
eigen_pairs.sort(reverse=True)
eigen_pairs[:]
Out[256]:
In [257]:
# print out eigenvalues
print('Eigenvalues in descending order: \n%s' %eigenvalues)
In [258]:
tot = sum(eigenvalues)
var_exp = [( i /tot ) * 100 for i in sorted(eigenvalues, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
plt.plot(var_exp)
Out[258]:
In [259]:
# Ploting
plt.figure(figsize=(8 , 7))
plt.bar(range(1, eigenvalues.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eigenvalues.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
In [260]:
# Reducing from 17 to 10 dimension space
pca = PCA(n_components=10)
data_reduced = pca.fit_transform(X_z)
data_reduced.transpose()
Out[260]:
In [261]:
pca.components_
Out[261]:
In [262]:
X_comp = pd.DataFrame(pca.components_,columns=list(X_z))
X_comp.head()
Out[262]:
In [263]:
# P_reduce represents reduced mathematical space.
# Reducing from 17 to 10 dimension space
P_reduce = np.array(eigenvectors[0:10])
# projecting original data into principal component dimensions
X_std_10D = np.dot(X_z,P_reduce.T)
# converting array to dataframe for pairplot
Proj_data_df = pd.DataFrame(X_std_10D)
In [264]:
#Let us check it visually
sns.pairplot(Proj_data_df, diag_kind='kde');
- Now, there are almost no correlation between independent attributes but there are some attributes which shows some correlation.
- The reason behind this is that some attributes in data are less correlated but we still taken that for dimentionality reduction.
- The solution may be we can remove columns which are less correlated then apply PCA.
In [265]:
# Split X and y into training and test set in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(Proj_data_df,y, test_size = 0.3, random_state = 10)
Logistic Regression¶
In [266]:
model = LogisticRegression()
model.fit(X_train, y_train)
prediction = model.predict(X_test)
In [267]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ',model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
- The accuracy, precision and recall value is reduced after applying PCA because we have reduced the dimensions.
- Despite giving less accuracy, precision and recall, this model is better as it has taken into consideration the relationship between the independent varibales and reduced the columns which are highly correlated.
- This model is performing well compared to Naive Bayes'and Naive Bayes' k fold models.
In [268]:
resultsDf=pd.DataFrame({'Model':['Logistic'],'Accuracy': model.score(X_test , y_test)},index={'1'})
resultsDf=resultsDf[['Model','Accuracy']]
resultsDf
Out[268]:
Naive Bayes' Classifier¶
In [269]:
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X_train, y_train)
prediction = model.predict(X_test)
In [270]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
- This model is not performing well as compared to other models.
In [271]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['Naive Bayes'], 'Accuracy': model.score(X_test, y_test)},index={'2'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[271]:
Using k fold cross validation in Naive Bayes¶
In [272]:
#Use the Naive Bayes CLassifier with k fold cross validation
scores = cross_val_score(model, Proj_data_df, y, cv=10)
print(scores)
print('Average score: ', np.mean(scores))
In [273]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['Naive Bayes k fold'], 'Accuracy': np.mean(scores)},index={'3'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[273]:
- The model is not performing well as compared to other models.
Support Vector Classifier¶
In [274]:
clf = SVC()
clf.fit(X_train, y_train)
prediction = model.predict(X_test)
In [275]:
# check the accuracy on the training data
print('Accuracy on Training data: ',model.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', model.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
- This model is not performing well.
In [276]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['SVM'], 'Accuracy': model.score(X_test, y_test)},index={'4'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[276]:
Using Grid Search to tune model parameters¶
In [292]:
#Grid search to tune model parameters for SVC
from sklearn.model_selection import GridSearchCV
model = SVC()
params = {'C': [0.01, 0.1, 0.5, 1], 'kernel': ['linear', 'rbf']}
model1 = GridSearchCV(model, param_grid=params, verbose=5)
model1.fit(X_train, y_train)
print("Best Hyper Parameters:\n", model1.best_params_)
In [294]:
print(" Results from Grid Search " )
print("\n The best estimator across ALL searched params:\n",model1.best_estimator_)
print("\n The best score across ALL searched params:\n",model1.best_score_)
print("\n The best parameters across ALL searched params:\n",model1.best_params_)
B. Share best Parameters observed from above step.
The best estimator across ALL searched params: SVC(C=0.1, kernel='linear')
The best score across ALL searched params: 0.9375587523144852
The best parameters across ALL searched params: {'C': 0.1, 'kernel': 'linear'}
In [278]:
#Build the model with the best hyper parameters
model = SVC(C=0.5, kernel="linear")
scores = cross_val_score(model, Proj_data_df, y, cv=10)
print(scores)
print(np.mean(scores))
In [279]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['SVM k fold'], 'Accuracy': np.mean(scores)},index={'5'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[279]:
- We can see that Logistic Regression and SVM k fold is given better results as compared to other models so we can use either of the two models to predict the silhouette as one of three types of vehicle.
In [280]:
#splitting the data in test and train sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 10)
In [281]:
# scaling the data using the standard scaler
from sklearn.preprocessing import StandardScaler
X_train_sd = StandardScaler().fit_transform(X_train)
X_test_sd = StandardScaler().fit_transform(X_test)
In [282]:
# generating the covariance matrix and the eigen values for the PCA analysis
cov_matrix = np.cov(X_train_sd.T) # the relevanat covariance matrix
print('Covariance Matrix \n%s', cov_matrix)
#generating the eigen values and the eigen vectors
e_vals, e_vecs = np.linalg.eig(cov_matrix)
print('Eigenvectors \n%s' %e_vecs)
print('\nEigenvalues \n%s' %e_vals)
In [283]:
# the "cumulative variance explained" analysis
tot = sum(e_vals)
var_exp = [( i /tot ) * 100 for i in sorted(e_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
In [284]:
# Plotting the variance expalained by the principal components and the cumulative variance explained.
plt.figure(figsize=(10 , 5))
plt.bar(range(1, e_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, e_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
In [285]:
eigen_pairs = [(np.abs(e_vals[i]), e_vecs[:,i]) for i in range(len(e_vals))]
eigen_pairs.sort(reverse=True)
eigen_pairs[:5]
Out[285]:
In [286]:
# generating dimensionally reduced datasets
w = np.hstack((eigen_pairs[0][1].reshape(-1,1), eigen_pairs[1][1].reshape(-1,1)))
print('Matrix W:\n', w)
X_sd_pca = X_train_sd.dot(w)
X_test_sd_pca = X_test_sd.dot(w)
In [287]:
X_train_sd.shape, w.shape, X_sd_pca.shape, X_test_sd_pca.shape
Out[287]:
In [288]:
from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train_sd, y_train)
print ('Before PCA score', clf.score(X_test_sd, y_test))
clf.fit(X_sd_pca, y_train)
print ('After PCA score', clf.score(X_test_sd_pca, y_test))
In [289]:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train_sd, y_train)
print ('Before PCA score', model.score(X_test_sd, y_test))
model.fit(X_sd_pca, y_train)
print ('After PCA score', model.score(X_test_sd_pca, y_test))
In [290]:
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X_train_sd, y_train)
print ('Before PCA score', model.score(X_test_sd, y_test))
model.fit(X_sd_pca, y_train)
print ('After PCA score', model.score(X_test_sd_pca, y_test))
5. Data Understanding & Cleaning:¶
5.A. Explain pre-requisite/assumptions of PCA.
- There must be linearity in the data set, i.e. the variables combine in a linear manner to form the dataset. The variables exhibit relationships among themselves.
- PCA assumes that the principal component with high variance must be paid attention and the PCs with lower variance are disregarded as noise. Pearson correlation coefficient framework led to the origin of PCA, and there it was assumed first that the axes with high variance would only be turned into principal components.
5.B. Explain advantages and limitations of PCA.
Advantages of PCA:
- Highlight similarities and differences in data
- Reduce dimensions without much information loss
- Reduces Overfitting
- Improves visualization
limitations of PCA:
- PCA is not robust against outliers: Similar to the point above, the algorithm will be biased in datasets with strong outliers. This is why it is recommended to remove outliers before performing PCA.
- PCA assumes a correlation between features: If the features (or dimensions or columns, in tabular data) are not correlated, PCA will be unable to determine principal components.